Direct memory access (DMA) is an efficient means for transferring data to and from a memory without direct involvement of a central processing unit (CPU). A DMA engine performs the desired data transfer operations as specified by DMA instructions, known as descriptors. The descriptors typically indicate, for each operation, a source address from which to read the data, and information regarding disposition of the data. The descriptors are commonly organized in memory as a linked list, or chain, in which each descriptor contains a field indicating the address in the memory of the next descriptor to be executed.
In order to initiate a chain of DMA data transfers, an application program running on a CPU prepares the appropriate chain of descriptors in a memory accessible to the DMA engine. The CPU then sends a message to the DMA engine indicating the memory address of the first descriptor in the chain, which is a request to the DMA engine to start execution of the descriptors. The application typically sends the message to the “doorbell” of the DMA engine—a control register with a certain bus address that is specified for this purpose. Sending such a message to initiate DMA execution is known as “ringing the doorbell” of the DMA engine. The DMA engine responds by reading and executing the first descriptor. The engine follows the “next” field through the linked list until execution of the descriptors is completed or terminated for some other reason.
DMA is used in modern network communication adapters to interface between host computer systems and packet networks. In this case, the host prepares descriptors defining messages to be sent over the network and rings a doorbell of the communication adapter to indicate that the descriptors are ready for execution. The descriptors typically identify data in the host system memory that are to be inserted in the packets. During execution of the descriptors, a DMA engine in the adapter reads the identified data from the memory. The adapter then adds appropriate protocol headers and sends packets out over the network corresponding to the messages specified by the descriptors.
Packet network communication adapters are a central element in new high-speed, packetized, serial input/output (I/O) bus architectures that are gaining acceptance in the computer industry. In these systems, computing hosts and peripherals are linked together by a switching network, commonly referred to as a switching fabric, taking the place of parallel buses that are used in legacy systems. A number of architectures of this type have been proposed, culminating in the “InfiniBand™” (IB) architecture, which is described in detail in the InfiniBand Architecture Specification, Release 1.0 (October, 2000), which is incorporated herein by reference. This document is available from the InfiniBand Trade Association at www.infinibandta.org.
A host connects to the IB fabric via a network adapter, which is referred to in IB parlance as a host channel adapter (HCA). When an IB “consumer,” such as an application process on the host, needs to open communications with some other entity via the IB fabric, it asks the HCA to provide the necessary transport service resources by allocating a transport service instance, or queue pair (QP), for its use. Each QP has a send queue and a receive queue and is configured with a context that includes information such as the destination address (referred to as the local identifier, or LID) for the QP, service type, and negotiated operating limits. Communication over the fabric takes place between a source QP and a destination QP, so that the QP serves as a sort of virtual communication port for the consumer.
To send and receive communications over the IB fabric, the consumer initiates a work request (WR) on a specific QP. There are a number of different WR types, including send/receive and remote DMA (RDMA) read and write operations, used to transmit and receive data to and from other entities over the fabric. WRs of these types typically include a gather list, indicating the locations in system memory from which data are to be read by the HCA for inclusion in the packet, or a scatter list, indicating the locations in the memory to which the data are to be written by the HCA. When the client submits a WR, it causes a work item, called a work queue element (WQE), to be placed in the appropriate queue of the specified QP in the HCA. The HCA then executes the WQE, including carrying out DMA operations specified by the gather or scatter list submitted in the WR. In this way, it generates outgoing packets and processes incoming packets so as to communicate with the corresponding QP of the channel adapter at the other end of the link.
WRs can thus be regarded as descriptors, specifying DMA and other operations to be executed by the HCA. In typical implementations, to initiate a WR, the consumer writes the corresponding descriptor to the system memory of the host and then rings a doorbell on the HCA. A range of addresses in the memory space of the host is assigned to the HCA for use as doorbells by consumers in accessing their allocated QPs. When the consumer writes to its assigned doorbell, it prompts the HCA to read the descriptor from the memory and service the WR.